A Transparent Grid Filesystem

Here we argue that the existence of a transparent grid filesystem greatly simplifies the user’s experience of grid middleware, particularly regarding data management. We focus on the EGEE grid environment, but we assert that the arguments are substantively true for arbitrary middlewares. We describe a prototype of a transparent grid filesystem, gridFS, with basic but relatively complete functionality provided by four abstract engines: directory, discovery, data movement, and consistency engines. This grid filesystem can be deployed on every node of EGEE, Globus, or any other existing middleware and even on user’s workstations. The grid filesystem is specifically intended to support interoperability between arbitrary middlewares. The data movement engine consists of the basic client and server sides of a filesystem. Its functionality is focussed on accessing various types of data storages, and operation at the block rather than at file level. The server side of the data movement engine exploits an enhanced version of the GridSite [26] module for the Apache webserver [27]. GridSite includes very desirable features including authentication and GACL [15] authorisation at each directory level, and supports convenient graphical editing of these permissions. Each directory contains a XML file that defines the permissions, conditioned by host, VO or person, based on the Globus Security Infrastructure (GSI) [28]. The client side of the data movement engine uses block-level cacheing to minimise traffic and support consistency. It supports a user-specific grid filesystem view. It uses HTTPS to communicate with the server-side, so it can traverse firewalls as easily as any browser can. By operating at block level, it can fractionally read or write remote files, only transferring the blocks in question. Consistency semantics define the outcome of multiple accesses to a single file. An example case where inconsistency may arise is when several grid users access